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FOREWORD 

This  study  was  Initiated  in  response  to  a request  from  the  Chief  of  Naval 
Personnel  (Pers-B2212,  now  Pers-5212)  to  develop  selection  procedures  that 
identify  petty  officers  who  have  the  potential  to  provide  counseling  services 
to  their  shipmates. 

The  Navy  Career  Counselor  rating,  which  was  established  in  1973,  is  one 
of  the  Navy  ratings  that  has  no  requirements  for  members  in  the  lower  grades. 

The  Navy  must  rely  on  inputs  from  other  ratings  to  achieve  and  maintain  re- 
quired levels.  The  skills  and  experiences  with  which  petty  officers  have 
demonstrated  competence  to  advance  in  their  occupational  specialty  may  not 
be  related  to  the  necessary  skills  of  effective  counseling.  Thus,  in  the 
absence  of  relevant  data  in  the  records  of  applicants,  the  emphasis  in 
this  study  was  to  develop  measures  and  selection  procedures  directly  relevant 
to  counseling  characteristics.  (Another  study  nearing  completion  is  addressed 
to  the  development  of  selection  procedures  to  identify  a general  characteristic 
of  Interpersonal  effectiveness,  as  a necessary  behavioral  quality  for  all  Navy 
ratings  that  involve  providing  face-to-face  services  to  shipmates.)  The  results 
of  this  study  are  considered  to  be  of  interest  and  relevance  to  other  selection 
requirements  for  jobs  involving  counseling  activities,  particularly  that  of 
recruiter. 

The  substantial  and  valuable  assistance  of  the  following  persons  is  grate- 
fully acknowledged:  Mr.  David  J.  Morena,  for  data  processing  and  computation, 

and  Ms.  Hazel  F.  Schwab,  for  clerical  support. 

This  study  was  performed  in  support  of  Exploratory  Development  Task 
Area  ZF55-521.03  (Career  and  Occupational  Design). 


J.  J.  CLARKIN 
Commanding  Officer 
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SUMMARY 


Problem 


Although  the  Navy's  training  and  career  programs  are  extensively  published, 
it  is  difficult  for  an  individual,  especially  one  on  a first  enlistment,  to 
become  aware  of  the  many  opportunities.  Thus,  the  Navy's  Career  Counseling 
Program  assigns  experienced,  knowledgeable  senior  petty  officers  to  assist 
personnel  in  taking  advantage  of  the  opportunities  relevant  to  the  individual's 
qualifications.  Such  service  is  essential  for  the  Navy  to  compete  with  other 
employers  to  attract  and  retain  quality  personnel. 

Purpose 

Selection  procedures  are  needed  to  identify  those  senior  petty  officers 
who  will  be  most  concerned  and  effective  in  providing  career  guidance  services. 
Since  cognitive  tests,  such  as  the  Navy  Basic  Test  Battery  (BTB) , are  usually 
poor  predictors  of  on-job  performance  or  Interpersonal  skills,  a battery- of 
noncognltlve  instruments  was  administered  to  Navy  petty  officers  performing 
career  counselor  duties  to  determine  whether  counseling  effectiveness  could 
be  predicted. 

Approach 

The  criterion  used  in  the  research  was  the  counselee's  perception  of  the 
quality  of  the  counselor's  immediate  interaction  with  the  counselee  (as  dis- 
tinguished from  an  ultimate  criterion  such  as  the  counselee's  reenlistment 
decision).  Thus,  criterion  data  were  acquired  directly  from  the  counselees 
who  evaluated  such  counselor  behaviors  as  pleasantness,  thoroughness,  and 
manifested  Interest  in  the  counselee's  concerns.  The  tests  administered  to 
counselors  were  Guilford  Tests  of  Social  Intelligence  (GTSI),  Comrey  Person- 
ality Scales  (CPS) , Strong  Vocational  Interest  Blank  (SVIB) , Dole  Ideal 
Counselor  Adjective  Check  List  (ICAC),  and  a locally  developed  Biographical 
and  Attitudlnal  Inventory  (BAI).  Scoring  keys  to  predict  counselor  effective- 
ness were  empirically  constructed,  using  the  counselee  evaluations  as  the 
criterion  data.  Standard  keys  for  the  BTB,  GTSI,  and  CPS  were  also  validated. 
Counselor-counselee  dyads  were  formed  by  age  and  BTB  score  subgroups  to 
determine  the  relationship  of  age  and  cognitive  aptitude  to  effective  counseling. 

Findings 

Generally,  counselees  evaluated  favorably  the  services  of  the  senior  petty 
officers  performing  counselor  duties,  and  counselee's  evaluations  were  useful 
to  validate  noncognltlve  selection  instrtments. 

1.  The  counselees  evaluated  the  counselors  quite  favorably  on  such  counselor 
characteristics  as  pleasantness,  understanding,  concern,  and  awareness.  Most 
counselees  Indicated  that  the  counselor's  help  enabled  them  to  find  out  or  do 
considerably  more  than  that  which  they  could  have  done  on  their  own.  Very  few 
counselees,  usually  less  than  10  percent,  evaluated  the  counselor's  assistance 
negatively. 

2.  The  counselor's  age  was  most  strongly  related  to  the  evaluations  of 
the  youngest  counselees  who  evaluated  32-34  year  age  counselors  higher  than 
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other  younger  and  older  counselors.  Neither  the  counselor's  pay  grade  nor 
aptitude  (Basic  Test  Battery)  scores  were  related  to  the  effectiveness  and 
thoroughness  of  the  counselor's  services.  However,  low  aptitude  counselees 
evaluated  the  counselor's  Helpfulness  more  highly  than  did  the  high  aptitude 
counselees . 

3.  The  validities  of  cognitive  type  aptitude  tests,  such  as  the  subtests 
of  the  Navy  Basic  Test  Battery,  were  near  zero.  Validities  of  standard  keys 
of  self-assessment  Instruments  varied  from  near  zero  to  the  low  20s.  Although 
low,  the  validities  near  .20  were  highly  slgnif leant. 

4.  The  validities  of  the  constructed  keys— those  which  were  specially  con- 
structed on  the  present  process-type  criterion  of  the  counselee's  perception 

of  the  counselor's  awareness  and  concern  (i.e.,  "warmth") —ranged  from  the  30s 
to  the  60s.  On  cross-validation,  most  of  the  validities  shrank  to  low  or  non- 
significant values.  Those  cross-validities  (near  .40)  that  remained  signifi- 
cant were  higher  than  the  validities  of  the  standard  keys. 

5.  Dependent  upon  selection  ratio,  use  of  the  standard  or  empirically  con- 
structed keys  would  Increase  the  percentage  of  superior  counselors  selected. 

It  was  demonstrated  that,  for  selection  ratios  ranging  from  30  to  70  percent, 
use  of  various  empirically  validated  keys  would  yield  proportionate  improve- 
ments from  8 to  26  percent. 

Conclusions 

Criterion  data  acquired  directly  from  counselees  were  quite  useful  to 
develop  counselor  selection  instruments.  The  use  of  noncognitlve  instruaients 
would  increase  the  percentage  of  superior  counselors  selected.  Of  the  con- 
structed keys,  the  best  cross-validated  single  and  multiple  selectors  were  the 
CPS  (_r  “ .34,  2.  ^ .006)  and  BAI-ICAC  composite  (r^  ” .41,  < .014)  respectively. 

Recoautendat  ions 

It  is  recommended  that: 

1.  The  CPS  be  used  as  one  of  the  selectors  for  career  counselors. 

2.  The  BAl  and  ICAC  be  administered  but  not  used  as  selectors  (since 
cross-validation  samples  for  these  two  multiple  selectors  were  only  half  that 
of  the  CPS),  and  that  all  three  instruments  be  revalidated  on  additional  sample 
data. 

3.  The  usefulness  of  the  three  Instruments  for  other  jobs  Involving  coun- 
seling activities  (e.g.,  recruiter)  be  investigated. 
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INTRODUCTION 


Problem  and  Background 


In  the  present  All-Volunteer  Force  environment,  the  military  services 
must  compete  in  the  labor  market  with  all  other  employers  to  attract  and 
retain  quality  personnel.  The  attractions  that  the  Navy  has  to  offer  are 
many  and  varied,  including  superior  technical  and  academic  training,  well 
structured  career  progressions  up  through  the  enlisted  and  officer  ranks, 
family  health  and  survivor  benefits,  and  a very  favorable  retirement  plan. 

The  details  of  these  attractions  and  the  eligibility  requirements  to 
qualify  for  them  are  described  extensively  in  a number  of  publications  and 
manuals.  However,  the  very  extent  of  the  details  makes  it  quite  difficult 
for  the  individual  Navy  person,  especially  one  who  is  on  a first  enlistment 
with  limited  administrative  experience,  to  be  fully  nware  of  available 
opportunities  for  a particular  person's  qualifications  and  circumstances. 

To  assist  Navy  personnel  in  taking  advantage  of  these  opportunities,  the 
Navy  Career  Counseling  Program  was  established. 

Guidance  for  administration  of  counseling  services  is  described  in  the 
Navy  Career  Counseling  Manual  (NAVPERS  15878-B) . Counseling  jobs  at  two 
levels  of  activity  are  described — full-time  (command;  and  part-time  (depart- 
ment/division). Assignment  to  the  full-time  counseling  jobs  is  controlled 
by  the  Bureau  of  Naval  Personnel.  Senior  petty  officers  in  the  occupational 
specialty  (Navy  "rating")  of  Navy  Career  Counselor  (NC)  and  also  in  other 
technical  ratings  are  eligible  for  consideration  for  these  assignments. 

Unit  and  staff  commands  are  encouraged  to  assign  part-time  counselor  responsi- 
bilities to  an  individual  in  each  division  or  department. 

The  tasks  of  the  career  counselor,  as  specified  in  the  Counseling  Manual, 
Include  the  following:  (1)  organizing  and  executing  the  command  counseling 

program,  (2)  providing  counseling  services,  (3)  assessing  the  professional 
caliber  and  career  potential  of  interviewees,  (A)  assessing  feelings  and 
atmosphere  at  the  working  level,  and  initiating  suggestions  to  the  managerial 
level  and  to  the  Chief  of  Naval  Personnel  to  improve  retention,  (5)  providing 
information  to  counselees  on  current  and  new  programs  and  opportunities,  and 
(6)  coordinating  efforts  with  full-time  counselors  at  the  command  level  and 
part-time  counselors  at  the  depar tment/dlvlslon  level. 

Thus,  an  important  need  appears  to  exist  for  the  services  of  a person  who 
is:  (1)  knowledgeable  of  the  details  of  programs  and  benefits,  (2)  highly 

skilled  in  communicating  the  information  most  relevant  to  an  Individual,  and 
(3)  helpful  in  assisting  the  individual  to  take  advantage  of  the  opportunities. 
The  skills  necessary  to  provide  these  services  may  not  be  related  to  the  skills 
in  which  petty  officers  have  demonstrated  competence  to  advance  in  their 
occupational  specialty.  The  problem  addressed  in  the  present  study  concerns 
how  to  identify  the  persons  who  will  be  most  concerned  and  effective  in  terms 
of  the  quality  of  the  interactions  that  take  place  between  the  counselor  and 
counselee  while  the  former  performs  the  tasks  discussed  above. 
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Present  qualifications  for  entry  into  full-time  counseling  jobs  specify 
two  objectively  measurable  criteria — petty  officers  of  at  least  Grade  E-6 
(or  E-5  serving  on  at  least  a second  enlistment  and  eligible  to  compete  for 
E-6);  and  no  record  of  disciplinary  or  personal  problems.  All  other  criteria 
are  judgmental,  including  demonstrated  judgment,  initiative,  and  motivation 
to  perform  counseling;  and  effective  and  persuasive  use  of  the  English 
language,  both  verbal  and  written. 

Some  types  of  instruments  useful  and  presently  available  to  select  candi- 
dates for  entry  or  advancement  in  the  counselor  rating,  in  terms  of  the  six 
counselor  tasks  listed  above,  are  supervisory  evaluations  for  Tasks  1,  3,  4, 
and  6,  and  an  objectively  scored  technical  knowledge  test  on  the  content  of 
the  career  opportunity  program  for  Task  5.  However,  for  the  very  heart  of  the 
program,  the  counseling  services  in  Task  2,  the  need  for  empathlc  sensitivity 
in  the  private  (one-to-one)  counseling  relationship  is  not  deemed  to  be  amen- 
able to  assessment  by  supervisors  or  knowledge  tests.  Thus,  the  criterion  of 
special  Interest  in  this  study  is  that  of  the  reaction  of  the  counselee  to  the 
interaction  with  the  counselor. 

Although  cognitive  tests  (e.g.,  the  Navy  Basic  Test  Battery — BTB — have  been 
highly  effective  in  predicting  technical  school  performance  (Thomas,  1972), 
such  tests  are  usually  found  to  be  poor  predictors  of  on-job  performance 
(Curtis,  1971).  For  jobs  that  Include  some  aspects  of  interpersonal  activities, 
noncognltive  tests  have  sometimes  demonstrated  moderate  validity  with  a cri- 
terion of  training  or  school  performance,  but  not  with  on-job  performance. 

For  example,  the  Yeoman  Key  of  the  Navy  Vocational  Interest  Inventory  (NVII) 
cross-validated  .23  on  a criterion  of  Final  School  Grade  (FSG)  for  the  Navy 
basic  Personnelman  "A*'  school,  compared  to  validities  of  -.02  for  the  General 
Classification  Test  (GCT)  and  -.01  for  the  Arithmetic  Test  (ARI)  (Thomas,  1970). 
In  contrast,  on  a criterion  of  job  performance  6 months  after  graduation  from 
the  Personnelman  school,  the  NVII  YN  Key,  the  Clerical  Test  (CLER) , and  some 
experimental  memory  tests  were  not  related  (Thomas,  1971).  GCT  and  ARI  corre- 
lated .14  -.19.  As  predictors,  peer  ratings,  instructor  ratings,  and  FSGs 
(measures  that,  of  course,  would  not  become  available  until  completion  of  the 
Personnelman  training)  correlated  .23  -.35.  Thomas  (1971)  concluded  that  it 
appears  that,  of  the  tests  that  are  or  could  be  administered  prior  to  school 
assignment,  only  the  BTB  tests  currently  used  in  the  PN  school  selection  corre- 
lated significantly,  though  not  very  highly,  with  on-job  performance  evaluations. 

Purpose 

A battery  of  noncognltive  Instruments  was  administered  to  Navy  petty  officers 
performing  career  counselor  duties  to  determine  whether  the  quality  of  the 
counseling  interaction  could  be  predicted.  Since  the  lack  of  good  criterion  data 
may  have  depressed  validities  in  other  studies  (e.g.,  Lau  & Abrahams,  1970), 
special  effort  was  directed  towards  developing  and  administering  counselee- 
relevant  criterion  measures. 
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Questions  specifically  addressed  in  the  present  study  are: 

1.  What  is  the  counselee's  reaction  to  the  interaction  with  the  counselor? 

2.  Are  couiiselor-counselee  differences  in  aptitude,  as  measured  by  the  GCT 
and  ARI  subtests  of  the  BTB,  reflected  in  the  counselee's  perceptions  of  the 
interaction? 

3.  Does  the  counselee  identify  best  with  a counselor  closest  to  the 
counselee's  own  age? 

4.  Are  aptitude,  personality,  vocational  interest,  and  biographical  instru 
ments  useful  in  predicting  the  counselor's  warm,  personal  concern  in  the  inter- 
action with  the  counselee? 


METHOD 


Instrumentation 


Criteria 


Measures  were  designed  to  provide  an  evaluation  of  counselor  services 
directly  from  the  recipient  of  the  services — the  counselee.  Statements  that 
described  various  aspects  of  counselor  behavior  (i.e.,  pleasantness,  manifested 
interest  in  counselee's  concerns,  openness  to  listening  to  counselee's  questions, 
thoroughness)  were  designed  for  counselee  responses  on  Likert-type  6-point 
scales.  The  complete  counselee  instrument  contained  a total  of  86  items, 
including  some  on  subjects  discussed  during  counseling,  job  satisfaction,  and 
occupational  interests.  Figure  1 displays  the  12  criterion  items  used  in  the 
present  study,  which  have  been  renumbered  consecutively  from  where  they  were 
Interspersed  in  the  complete  instrument. 
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Each  of  Che  stacemerts  balov  appllae  to  this 
Counaelor.  Circle  the  code  number  which  tella 
Che  extent  of  your  agreement  or  dlaagraement 
with  each  statement  aa  It  appllea  to  the  asala- 
tance  you  received. 


Thia  Counaelor; 

Waa  very  aware  and  appreciative  of  my 
interests  or  plans. 

Waa  very  pleasant. 

Treated  me  like  a number  rather  than  a person. 

Understood  how  I felt. 

Made  me  feel  comfortable  and  at  eaae. 

Cave  me  plenty  of  opportunities  to  ask  questions. 

Acted  as  If  my  concerns  were  important. 

Was  thorough  and  effectively  solved  my  problem 
or  provided  the  desired  service. 

Helped  me  to  see  clearly  what  I had  to  do  to 
cake  care  of  my  problem 
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(.'offlpared  to  what  you  think  you  could  have  found 
out  or  done  through  your  own  efforts,  how  much 
help  do  you  think  Uil^  Counselor  was. 
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Figure  1. 


Item 

-X. 

Abbrev 

This  Counselor  was  helpful  In: 

10 

INFRMC 

Informing  you  of  Navy  programs  and  benefits  of 
advantage  to  vou. 

11 

BENFIT 

Working  out  the  ways  for  you  to  take 
order  to  lake  advantage  of  any  Navy 
or  benefits. 

action  in 
programs 

12 

PRSPRB 

Working  out  a solution  to  a personal 
or  obstacle. 

problam 

Criterion 

items  from  counselee's 

evaluation  of  counselor. 
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Predictors 


The  batteries  of  tests  and  Inventories  that  were  evaluated  are 
described  briefly  below. 

1.  The  Navy  Basic  Test  Battery  (BTB)  of  aptitude  tests — usually 
administered  to  new  recruits,  and  used  primarily  to  select  students  for  basic 
technical  training. 

a.  General  Classification  Test  (GCT) — Verbal  analogy  and  sentence 
completion  (100  Items). 

b.  Arithmetic  Test  (ARI) — Arithmetic  compu*‘'>tlon  (20  Items)  and 
reasoning  (30  Items) . 

c.  Mechanical  Test  (MECH) — Tool  knowledge  and  mechanical  compre- 
hension (50  Items) . 

d.  Clerical  Test  (CLER) — Highly  speeded  number-matching  for  the 
ability  to  observe  quickly  and  accurately  (210  Items). 

e.  Sonar  Pitch  Memory  Test  (SONAR) — Recorded  auditory  signals 
for  the  ability  to  discriminate  small  pitch  differences  (40  Items). 

f.  Radio  Code  Aptitude  Test  (RADIO) — Recorded  Morse  Code  (150 

Items) . 

g.  Electronics  Technician  Selection  Test  (ETST) — Measures  of 
achievement  and  knowledge  related  to  electronic  maintenance.  Including  mathe- 
matics, science,  shop  practice,  electricity,  and  radio  (80  Items). 

2.  Guilford  Tests  of  Social  Intelligence  (GTSI) — Only  three  of  the 
available  six  subtests  were  used. 

a.  Missing  Pictures — Interpretation  of  social  Interactions 
(depicted  by  a sequence  of  photographs)  by  selection  of  one  of  three  alter- 
native photographs  that  completes  the  sequence  (20  Items). 

b.  Cartoon  Predictions — Interpretation  of  Intentions  or  feelings 
from  facial  expressions,  body  postures,  and  situational  circumstances  de- 
picted by  cartoon  characters  (30  Items). 

c.  Social  Translations — Detection  of  subtle  differences  In 
verbal  communications  that  Imply  different  Intentions  or  meanings  (24  Items). 

3.  Comrey  Personality  Scales  (CPS) — Measures  of  frequency  of  partici- 
pation or  Intensity  of  feelings  In  specified  events,  experiences,  situations, 
or  outlooks  (180  Items).  Scoring  keys  have  been  developed  to  measure  eight 
bipolar  traits  plus  validity  and  response  bias.  The  first  terms  of  each  trait, 
as  used  In  this  study  as  titles,  are  trust,  orderliness,  conformity,  activity, 
stability,  extroversion,  masculinity,  and  empathy. 
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4.  Strong  Vocational  Interest  Blank  (SVIB) — A self-report  inventory 
of  academic,  occupational,  recreational,  and  athletic  Interests  (399  Items). 
Scoring  keys  have  been  developed  for  a great  number  of  occupations  and  for  a 
moderate  number  of  traits. 

5.  Ideal  Counselor  Adjective  Check  List  (ICAC) — An  inventory  of  215 
adjectives  developed  by  Dole  (1964)  (43  items).  The  instructions  were  modi- 
fied slightly  to  accommodate  the  electronically  scannable  Optical  Mark  Reader 
(OMR)  response  sheets  used  In  the  present  duty. 

6.  Biographical  and  Attltudlnal  Inventory  (BAI) — A self-report 
Inventory  containing  biographical  Items  of  the  type  advocated  by  Owens  (1976), 
including  Items  about  high  school  experiences  and  social  and  family  activ- 
ities;^ and  also  containing  attltudlnal  Items  developed  locally  (120  Items). 

Sampling  Procedure 

Counselors  were  middle  and  senior  grade  enlisted  petty  officers  who  were 
assigned  counseling  duties  on  either  a full-  or  part-time  basis.  Sets  of 
questionnaires  to  be  completed  by  counselees  (the  criterion  instrument)  were 
forwarded  via  the  unit  command  to  the  counselors,  with  Instructions  that  re- 
quested the  counselor  to  Issue  a counselee  questionnaire  letter  to  the  next 
consecutive  eight  counselees  Interviewed  by  a full-time  counselor  or  the 
next  six  by  a part-time  counselor.  Counselees  were  requested  to  complete 
the  questionnaire  within  48  hours  after  the  Interview  and  mall  It  directly 
to  this  research  center  (In  a preaddressed  envelope  provided). 

Counselees  were  generally  In  the  Junior  pay  grades,  on  a first  enlistment 
for  which  a series  of  orientation  and  progress  interviews  Is  specified. 

After  a few  months  had  elapsed,  while  counselee  questionnaires  were  re- 
ceived, the  predictor  Instruments  2 through  6 described  above  were  forwarded 
to  all  counselors  for  whom  counselee  responses  had  been  received.^ 

The  final  number  of  usable  Instruments  available  was  as  follows:  (1)  for 

scoring  key  validation,  311  counselor  sets  (l.e.,  with  6 predictor  Instruments 
and  at  least  3 counselee  evaluations  per  counselor)  (see  Table  1);  and  (2)  for 
Item  Intercorrelations  and  counselor-counselee  relationships,  2,075  counselee 
evaluation  sets.  To  develop  realistic  and  stable  criteria  for  key  validation, 
the  data  were  acquired  (1)  directly  from  the  counselee  (vice  the  counselor's 
supervisor),  and  (2)  from  more  than  one  counselee  per  counselor.  The  number 
of  counselee  evaluations  per  counselor  ranged  from  126  counselors  with  one 
evaluation  each,  to  one  counselor  with  11  evaluations  (see  Table  2). 


^These  Items  were  taken  from  an  unpublished  Inventory,  "Booklet  B,  Auto- 
biographical Data,"  developed  by  Owens  (Note  1). 

^Scores  for  the  Navy  BTB  subtests  were  already  available  from  the  Enlisted 
Master  Tape  file. 
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Table  1 


Navy  Career  Counselor  and  Counselee 
Instrumentation  and  Sample  Sizes 


Counselor  Predictor  Packets  Counselee  Criterion  Packets 


Type 

Counselor 

Administered 

Returned^ 
and  Usable 

Administered'^ 

Q 

Returned 
and  Usable 

N 

N % 

N 

N 

% 

Full-Time 

412 

224 

54 

2800 

1514 

54 

Part-Time 

240 

87 

36 

1200 

561 

47 

Total 

652 

311 

48 

4000 

2075 

52 

^Of  406  returned,  95  were  incomplete  and  not  used. 

^Estimated  from  a Counselor’s  log  of  inteiT^iews  which  was  submitted 
by  the  Counselor. 

‘^Of  2365  returned,  290  were  incomplete,  or  particularly,  could  not 
be  matched  to  a Counselor  and  were  not  used. 


Table  2 

Distribution  of  Counselees  Per  Matched  Counselor 


N Counselees 
Identified  with 
a Counselor 

N Counselors® 
Matched  with 
Counselees 

Total 

Counselees 

11 

1 

11 

9 

2 

18 

8 

21 

168 

7 

33 

231 

6 

46 

276 

5 

58 

290 

4 

76 

304 

3 

89 

267 

2 

150 

300 

1 

126 

126 

Total 

602 

1991 

*Full  and  part-time  Counselors  combined. 


8 


Analysis  of  Counselee  Evaluation  Items 


The  scale  points  on  the  Llkert-type  scales  were  weighted  from  6 (most 
favorable)  to  1 (least  favorable).^  Means,  standard  deviations,  percentage 
endorsements,  and  Intercorrelatlons  were  calculated  for  the  total  group  of 
counselees  responding  to  each  of  the  12  Individual  criterion  Items. 

Construction  of  Criterion  Variables 


The  three  kinds  of  counselor  behavior  of  Interest  were  the  counselor's: 

(1)  demonstration  of  personal  concern  ("warmth")  In  Interaction  with  the 
counselee,  (2)  effectiveness  (as  perceived  by  the  counselee)  at  assisting 
the  counselee  In  solving  problems  specific  to  the  Interview,  and  (3)  demon- 
stration of  overall  helpfulness  (again  as  perceived  by  the  counselee). 

The  12  counselee  evaluation  Items  were  rationally  grouped,  by  homogeneity 
of  content  as  determined  by  multiple  judges,  to  form  three  separate  criterion 
variables  as  follows:  Warmth  (Items  1-7),  Problem-solving  Effectiveness 

(Item  8),  and  Helpfulness  (Items  9-12).  The  criterion  score  (on  each  Item 
set)  for  each  counselor  was  formed  by  first  calculating  an  Individual  counselee 
average  for  each  Item  set,  and  then  calculating  an  average  from  all  counselees 
evaluating  the  particular  counselor  (l.e.,  the  averages  of  the  multiple 
evaluations  on  each  Item  set  were  used  as  the  criterion  scores  for  each  coun- 
selor). The  analyses  performed  with  each  of  the  criterion  scores  are  de- 
scribed below. 

Interrater  Agreement  Among  Counselees 

To  measure  Interrater  agreement  among  the  stochastically  Independent 
counselee  evaluations,  the  Intraclass  Correlation  Coeflclent  (Pj)  was  cal- 
culated employing  the  formula  for  the  case  of  unequal  class  membership 
(Haggard,  1958,  although  Haggard  employs  the  symbol  R for  the  coefficient). 

The  levels  of  significance  of  the  obtained  coefficients  were  tested  by  the 
same  F-ratlo  which  was  computed  from  the  same  mean  squares,  with  the 
appropriate  df , as  were  used  to  obtain  the  coefficient  (Haggard,  1958). 

Various  groupings  of  counselees  per  counselor  were  Investigated  for  two  of 
the  three  criterion  Items:  (1)  for  the  single-item  Problem-solving  Effec- 

tiveness set — two  or  three,  at  least  four,  and  at  least  two  counselees  per 
counselor;  and  (2)  for  the  average  of  the  7-ltem  Warmth  set — at  least  two 
and  at  least  three  counselees  per  counselor. 

Counselor-Counselee  Aptitude  Differences 

An  analysis  of  variance  (ANOVA)  design  was  used  to  Identify  differences 
in  the  above  three  criterion  variables  related  to  counselor-counselee  dif- 
ferences In  aptitude,  as  measured  by  the  GCT  and  ARI  subtests  of  the  BTB. 
Counselor-counaelee  aptitude  dyads  were  formed  by  subgrouping  counselors 
Into  four  subgroups  by  the  sum  of  the  GCT+ARI  scores,  56-99,  100-110,  111- 
120,  and  121  and  above,  and  counselees  by  four  similar  subgroups.  Thus, 


^As  shown  In  Figure  1,  the  weights  were  reversed  for  Item  3. 
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16  subgroups  were  formed,  with  each  of  the  four  aptitude  groups  of  counselees 
evaluating  four  aptitude  groups  of  counselors.  Three  types  of  analyses  were 
performed  on  the  above  4x4  matrix  using  the  Warmth,  Problem-solving  Effec- 
tiveness, and  Helpfulness  criteria  as  dependent  variables.  A two-way  ANOVA 
was  performed  on  counselor-counselee  BTB  score  effects  and  interaction;  and 
a one-way  ANOVA,  on  counselor  BTB  effects  for  each  level  of  counselee  BTB. 

A posteriori  comparisons  of  pairs  of  means  were  performed  using  Duncan's 
Multiple  Range  Tests  (Overall  & Spiegel,  1969).  An  a priori  ^ test  was 
performed  on  three  selected  pairs  of  means  (i.e.,  the  maximum  number  of  pairs 
allowed  to  minimize  chance  results — see  Kirk,  1968).  Also,  in  a simpler 
analysis,  counselor  and  counselee  GCT+ARI  scores  were  simply  dichotomized 
at  a score  of  100,  and  counselee  evaluations  then  compared  on  the  three 
criteria. 

Counselor-Counselee  Age  Differences 

To  determine  whether  the  counselee  identifies  best  with  a counselor 
closest  to  the  counselee' s own  age,  counselor-counselee  age  dyads  were  formed 
by  subgrouping  counselors  into  four  age  groups — 21-31,  32-34,  35-37,  and  38 
and  above;  and  counselees,  17-19,  20-21,  22-23,  and  24  and  above.  Again,  16 
subgroups  were  formed,  with  each  of  the  four  age  groups  of  counselees  evalu- 
ating four  age  groups  of  counselors  (i.e.,  counselee  age  group  17-19  evalu- 
ating counselor  age  groups  21-31,  32-34,  35-37,  and  38  and  above;  counselee 
age  group  20-21  evaluating  other  independent  counselor  groups  of  above  age 
ranges;  etc.).  Employing  the  three  criteria,  the  same  three  types  of  analyses 
were  performed  as  were  described  above  for  the  BTB  score  dyads — two-way  and 
one-way  ANOVA,  and  a priori  comparisions  of  three  selected  pairs  of  means. 

Validation  of  Counselor  Predictors 


The  demonstration  of  a warm,  personal  concern  by  the  counselor  appears  to 
be  a voluntary,  discretionary  form  of  behavior  that  is  least  amenable  (1)  to 
supervisory  monitoring  during  a one-to-one  type  counseling  interaction,  and 
(2)  to  prediction  by  objective  testing.  Thus,  the  primary  emphasis  of  this 
study  was  to  evaluate  the  usefulness  of  the  predictors  to  select  candidates 
who  would  demonstrate  this  warmth  while  performing  the  counseling  services. 

Employing  the  Warmth  criterion,  the  total  group  of  usable  counselors 
(i.e.,  for  which  evaluations  were  available  from  at  least  three  counselees) 
was  used  for  an  independent  validation  of  the  standard  scores  for  predictor 
instruments  1,  2 and  3 described  above — 7 subtests  of  the  BTB,  3 subtests  of 
the  GTSI,  and  10  keys  of  the  CPS  test. 

Sltuatlonally  unique  scoring  keys  were  empirically  derived  for  predictors 
2-6,  using  two-thirds  of  the  available  counselor  group  for  key  construction 
(KEYCON)  and  the  other  third  as  a holdout  group  for  cross-validation. 

High  and  low  criterion  groups  were  formed  by  splitting  the  total  counselor 
group  at  the  median  of  the  criterion  score.  To  provide  a meaningful  multiple 
regression  analysis  when  various  combinations  of  predictors  were  tested,  as 
described  below,  the  same  criterion  groups  and  validation  groups  were  used 


for  all  predictors.  (As  an  alternative,  the  random  assignment  of  subjects 
to  the  KEYCON  or  holdout  groups  could  have  been  performed  separately  and 
repeatedly  for  the  construction  of  each  scale,  so  that  different  subjects 
could  have  been  randomly  selected  for  holdout  groups  for  different  predictors. 

Keys  were  constructed  for  GTSI,  SVIB,  ICAC,  and  BAI  by  selecting  the 
item  alternatives  (e.g.,  for  the  3 alternatives  of  each  item  of  the  SVIB, 
a total  of  1,197  alternatives  were  tested)  for  which  there  was  at  least  an 
8 percent  difference  in  responses  between  the  high  and  low  criterion  groups. 
Selected  alternatives  were  weighted  by  each  increment  of  eight  percentage 
points  difference  (e.g.,  a weight  of  2 for  at  least  16  percent  difference, 
and  3 for  24  percent  difference). 

For  the  CPS,  a different,  novel  key  construction  procedure  was  employed. 
Since  the  Likert-type  7-point  CPS  scale  was  longer  than  those  of  the  other 
predictors  (e.g.,  3-point  for  SVIB,  5-point  for  ICAC),  the  proportions  of 
endorsements  distributed  cunong  the  alternatives  were  relatively  small  and 
thus  unstable.  Also,  it  is  assumed,  for  the  standard  key  construction 
procedure,  that  alternatives  are  categorical.  Thus,  the  procedure  does 
not  take  advantage  of  the  additional  information  available  in  the  ordinal 
scale  of  the  CPS.  As  an  alternative,  a Tilton  Percentage  Overlap  statistic 
(Tilton,  1937)  between  the  high  and  low  criterion  groups  was  calculated  for 
the  distribution  of  responses  across  all  alternatives  of  each  item.  The 
items  (as  distinguished  from  alternatives)  with  at  least  an  8 percent  non- 
overlap  were  selected  for  the  scoring  key  (and  Incrementally  weighted  as 
described  above). 

After  high  and  low  criterion  groups  of  the  key  construction  and  cross- 
validation  groups  were  scored  by  each  key,  means,  standard  deviations, 
point  blserlal  coefficients,  and  Tilton  Percentage  Overlap  were  calculated. 
Stepwise  multiple  regression  analyses  were  performed  for  various  combinations 
of  predictors.  The  intercorrelation  matrices  for  multiple  regression 
equations  from  the  same  sample  on  which  key  construction  has  been  performed 
contain  validities  that  are  differentially  inflated  due  to  the  mix  of  standard 
and  constructed  keys,  length  of  tests,  and  number  of  alternatives.  Thus, 
the  regression  equations  may  yield  regression  weights  or  factors  that  are 
not  optimal  for  other  samples  (Abrahams  & Alf,  1972).  A preferred  pro- 
cedure, the  use  of  an  additional  sample  for  development  of  the  regression 
equations,  was  not  feasible  for  this  study  due  to  small  sample  sizes.  In 
addition  to  multiple  regressions  on  all  variables,  multiple  regressions 
on  pairs  of  variables  were  performed  and  the  pair  with  the  highest  cross- 
validation  coefficient  was  selected.  (A  further  cross-validation  of  these 
tests  and  regression  weights  on  another  sample  would  be  appropriate.) 

The  distribution  of  scores  obtained  from  cross-validation  of  the  con- 
structed CPS  key  was  used  to  construct  individual  expectancy  charts 
(Gulon,  1965;  Lawshe,  Bolda,  Brune,  & Auclalr,  1958)  that  predict  the 
probability  of  an  applicant  performing  as  a superior  Navy  career  counselor. 
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RESULTS 


Counselee  Evaluations  of  Counselor 

Generally,  counselees  evaluated  the  services  provided  by  the  counselor 
very  favorably.  For  the  6-polnt  Likert-type  scale  employed,  the  modal 
response  to  each  of  the  12  criterion  items  was  found  to  be  located  at  one  of 
the  top  two  scale  points  (see  Table  3).**  The  means  of  the  weighted  responses 
ranged  from  4.276  (Item  12 — Solution  to  Personal  Problems)  to  5.499  (Item  2 — 
Pleasantness)  (see  Table  4).  Generally,  counselors  were  evaluated  higher  on 
the  Warmth  set  (Items  1-7)  than  on  Helpfulness  (Items  9-12).  Most  counselees 
indicated  that  the  counselor's  help  was  considerably  more  than  that  which  the 
counselee  could  have  done.  Very  few  counselees,  usually  less  than  10  percent, 
evaluated  the  counselor's  assistance  negatively. 

The  Intercorrelations  among  the  criterion  items  ranged  from  .138  to  .663 
(see  Table  4).  The  Intercorrelation  patterns  do  not  clearly  reflect  the 
groupings  (into  the  three  criterion  sets)  rationally  determined  by  homogeneity 
of  content,  possibly  as  a result  of  the  negative  orientation  of  Item  3,  and  the 
different  scale  design  of  Items  10  through  12.  Interrater  agreement  among  the 
counselees  evaluating  each  counselor  was  rather  low,  but  highly  significant 
(see  Table  5,  e.g.,  for  the  7-ltem  Warmth  criterion,  using  all  evaluations  for 
three  or  more  counselees  per  counselor,  p^.  ■ .13,  £ < .001). 

Counselor-Counselee  Aptitude  Differences  and  Counselee  Evaluations 


In  the  comparison  of  the  counselor-counselee  BTB  (GCT+ARI)  dyads  on  the 
Helpfulness  criterion,  no  significant  differences  were  identified  among  the 
counselor  groups  (see  Table  6 and  Figure  2),  but  some  differences  were  iden- 
tified among  counselee  groups.  Generally,  the  low  BTB  counselees  evaluated 
the  counselor's  Helpfulness  more  highly  than  did  the  high  BTB  counselees 
(counselee  main  effect,  £ < .002). 

In  the  comparlslon  of  high  and  low  BTB  counselees  on  two  different  cri- 
terion items  (see  Figures  3 and  4),  the  high  BTB  counselees  evaluated  coun- 
selors higher  on  Warmth  (£  < .023),  but  lower  on  Helpfulness  (£  < .002), 
than  did  the  low  BTB  counselees.  Differences  among  BTB  subgroups  for  the 
Problem-solving  Effectiveness  criterion  (Item  8)  were  very  slight  and  non- 
significant. 

Counselor-counselee  Age  Differences  and  Counselee  Evaluations 


For  the  Problem-solving  Effectiveness  criterion  (Item  8),  the  counselor's 
age  was  found  to  be  related  most  strongly  to  the  evaluations  of  the  youngest 
counselees,  age  17-19  (see  Tables  7 and  8,  and  Figure  5).  (Table  8 and  the 
left-hand  side  of  Table  7 present  the  ANOVA  results,  while  the  right-hand 
side  of  Table  7 presents  the  results  of  the  differences  between  a priori 
selected  pairs  of  means.)  The  youngest  counselees  evaluated  the  youngest 


‘'Because  of  the  large  number  of  tables  and  figures  and  relatively  small 
amount  of  text  appearing  in  this  section,  the  tables  and  figures  are  pro- 
vided at  the  end  of  the  section,  in  the  order  that  they  are  referred  to  in 
text. 
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counselors  (age  21-31)  the  lowest;  and  the  age  32-34  counselors,  the  highest 
(a  posteriori  < .05,  a priori  £ < .01).  The  evaluations  of  the  youngest 
counselees  (age  17-19)  tended  to  be  curvilinear  (see  Figure  5)  with  the 
highest  evaluations  for  the  counselor  groups  in  the  middle  of  the  age  range; 
and  the  evaluations  of  the  oldest  counselees  (age  24  or  higher),  relatively 
flat.  However,  in  the  two-way  ANOVA,  the  differences  in  evaluations  among 
the  counselee  age  groups,  across  the  counselor  age  groups,  were  not  sig- 
nificant (see  footnote  of  Table  7,  i.e.,  for  interaction  effects,  < .381). 

Thus,  the  differences  in  slope  of  the  lines  among  the  counselee  age  groups 
were  not  significant,  although  the  differences  among  the  counselor  age  groups 
for  the  combined  evaluations  of  all  the  counselee  age  groups,  were  significant 
(i.e.,  2.  1 *001  for  main  effects  by  counselor  age  groups). 

For  the  Helpfulness  criterion,  differences  in  evaluations  among  counselor 
age  groups  were  significant  in  the  two-way  ANOVA  (£  < .036),  with  the  young- 
est counselors  evaluated  the  lowest  and  those  in  the  32-34  age  range,  the 
highest.  Counselee  age  differences  and  interaction  effects  were  not  sig- 
nificant for  the  Helpfulness  criterion. 

For  the  Warmth  criterion,  no  differences  in  the  two-way  ANOVA  were 
significant. 

Key  Construction  and  Validation 

The  validity  coefficients  of  the  keys  which  were  empirically  constructed 
from  the  criterion  of  primary  interest.  Warmth,  were  found  to  range  from 
.38  for  CPS  to  .66  for  SVIB  (see  Table  9).  On  cross-validation,  with  the 
typical  shrinkage  of  the  validity  coefficients,  only  the  CPS  key  remained 
significant  (r.  * *34,  < .05).  The  independent  validation  (on  the  total 

group  of  counselors)  of  various  standard  keys  yielded  validities  (see 
Table  10)  ranging  from  -.11  (Masculinity-CPS)  to  .22  (Activity-CPS) . 

Generally,  the  validities  of  the  cognitive  test  (e.g.,  BTB)  were  near  zero, 
and  the  noncognltlve  were  between  zero  and  the  20s.  Employing  various  i 

combinations  of  cognitive  and  noncognitive  tests  as  multiple  predictors 
(see  Table  11),  the  BAI  and  ICAC  yielded  a cross-validity  of  .41  (£  < .01), 
which,  with  the  addition  of  the  CPS-constructed  key,  dropped  to  .39  (p^  < .02). 

j 
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Table  3 


Distribution  of  Counselee  Responses  to  Criterion  Items 


Item 

No. 

Abbrev . 

Percentage  Endorsement 
Scale  Weights 
6 5 4 3 2 1 

Total 

N 

Responses 

1 

APRETV 

32 

47 

15 

4 

2 

- 

2046 

2 

PLESNT 

58 

37 

4 

- 

- 

- 

2046 

3 

NUMBER*’ 

58 

30 

4 

3 

3 

2 

2035 

4 

UNDFLT 

21 

51 

20 

5 

2 

1 

2040 

5 

CMFRBL 

45 

43 

10 

2 

1 

- 

2048 

6 

ASKQST 

56 

36 

6 

1 

- 

- 

2041 

7 

CONCRN 

38 

46 

12 

3 

1 

1 

2036 

8 

* PRBSLV 

28 

44 

18 

6 

3 

1 

2031 

9 

HLPPRB 

30 

45 

18 

5 

2 

1 

2026 

•10 

INFRMG 

36 

33 

22 

7 

1 

1 

1941 

11 

BENFIT 

29 

38 

23 

8 

1 

1 

1933 

12 

PRSPRB 

18 

28 

25 

21 

5 

2 

1397 

0 

Due  to  rounding,  percentages  do  not  necessarily  sum  to 

100. 

'^Note  that  scale  weights  were  reversed  for  this  item-- 
see  Figure  1. 
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Distribution  Statistics  and  Intercorrelations  Among  Criterion  Items 
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Note.  Decimal  points  for  correlations  have  been  omitted.  See  Figure  1 for  description  of  items. 


Table  5 


Interrater  Agreement  of  Counselees'  Evaluations 
Of  Counselors  for  Two  Criteria 


Criterion 

Item 

No. 

Source 

N of  Counselee 
Evaluations  per 
Counselor 

N of  Intraclass 
Groups  of 
Counselors 

Intraclass 

Coefficient^ 

Pi 

Problem-solving 

8 

2 or  3 

239 

.02 

NS 

Effectiveness 

8 

^ 4 

237 

.18 

.001 

8 

> 2 

476 

.23 

.001 

Warmth 

X of  1-7 

>.  2 

472 

.13 

.001 

X of  1-7 

^ 3 

326 

.13 

.001 

^Formula  for  unequal  class  membership  (Haggard,  1958:  Formula  5). 
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Standard  Deviations  of  Counselor  Helpfulness 
By  Counselor-Counselee  GCT+ARI  Dyads 


COUNSELOR  HELPFULNESS 


£ Test  Significant  Counselee 
GCT+ARI 


56-99 


100-110 


111-120 


121-Up 


COUNSELOR  GCT+ARI 


Figure  2.  Counselor  Helpfulness  by  counselor-counselee  GCT+ARI 
score  group  dyads. 


J 


WARMTH  (7 -ITEM  SET) 


< 100 


i 100 


Total 

(p:i.002) 


COUNSELOR  GCT+ARI 

Figure  4,  Counselee  evaluations  of  Counselor's  Helpfulness 
by  Basic  Test  Battery  (GCT+ARI)  score  dyads. 
(Total  Ns  are  greater  than  sum  of  dyad  Ns  because 
some  counselor  BTB  scores  were  not  available  to 
form  the  dyads.) 
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greater  for  total  than  subgroups  since  no  cases  were  deleted 


Table  8 


A Posteriori  Significance  Test  of  Counselor  Problem-Solving 
Effectiveness  by  Counselor-Counselee  Age  Dyads 


NS  - Not  significant. 
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COUNSELOR  PROBLEM  SOLVING  EFFECTIVENESS 


Figure  5.  Counselor  problem-solving  effectiveness  by  counselor- 
counselee  age  group  dyads. 
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Table  10 


Cross-Validities  of  Constructed  and  Standard  Keys 


Standard  Keys  (con't) 


For  constructed  keys,  N represents  the  one-third  holdout  group;  for  standard 
keys,  the  total  sample. 

**Criterion  measure  is  counselee  evaluation  from  7-item  Warmth  set. 
c 

Null  hypothesis  tested  for  standard  keys,  for  which  the  total  sample  was 
'available,  was  two-tail  r ■ 0;  and  for  constructed  keys,  one-tail  (see  Table  9). 
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Table  11 
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Multiple-Regression  Analysis  and  Empirical 
Cross-Validation  of  Counselor  Predictors 


Predictor 

Regression  Analysis^ 

Empirical 

Cross-Validity^ 

Abbreviation® 

Accretion 
of  JTth 
Variable 

R 

SE 

F 

P- 

r 

PI.® 

CPS  (Con) 

1st 

.38 

31.00 

1,168 

28.21 

.01 

.34 

.006 

BAI  (Con) 

1st 

.55 

28.14 

1,  40 

17.47 

.01 

ICAC  (Con) 

2nd 

.62 

26.68 

2,  39 

12.47 

.01 

.41 

.014 

BAI  (Con) 

1st 

.55 

27.93 

1,  98 

42.81 

.01 

ICAC  (Con) 

2nd 

.62 

26.28 

2,  97 

31.01 

.01 

CPS  (Con) 

3rd 

.63 

26.35 

3,  96 

20.73 

.01 

.39 

.022 

RADIO  (Std) 

1st 

.21 

32.79 

1,  64 

3.08 

NS 

SONAR  (Std) 

2nd 

.30 

32.32 

2,  63 

3.04 

NS 

CPS  (Std) 

3rd 

.36 

31.87 

3,  62 

3.01 

.05 

OCCUP'*  (Con) 

4th 

.41 

31.43 

4,  61 

3.00 

.05 

.34 

.126 

Con- -Constructed  Key,  Std — Standard  Key. 

^’criterion  is  7-item  Warmth  set. 

c 

Based  on  separate  one-third  holdout  group. 

^OCCUP--a  categorical  variable  which  identified  the  counselor’s  prior 
technical  specialty  in  one  of  four  occupational  areas--administrative,  opera- 
tor, mechanic,  or  all-other. 

Null  hypothesis  tested  was  two-tail  r = 0. 


I 
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DISCUSSION 


Interrater  Agreement  Among  Counselees 

It  was  reasonably  expected  that  interrater  agreement  would  not  be  high, 
since  the  counseling  sessions  are  conducted  on  a wide  variety  of  discussion 
topics  (e.g.,  promotion,  training  qualifications,  personal  problems,  etc.) 
and  under  a wide  range  of  environmental  conditions  (e.g.,  at  sea  vs.  In  port). 
Because  of  the  reality  of  the  variety  of  these  kinds  of  situations.  It  was 
deemed  appropriate  and  essential  to  employ  a criterion  from  several  counselees 
who  evaluated  the  counselor  under  these  various  conditions. 

Counselor-Counselee  Aptitude  and  Age  Differences 

It  was  also  expected  that  counselees  would  Identify  most  closely  with 
the  counselors  nearest  their  own  age,  and  also  evaluate  those  (youngest)  coun- 
selors the  highest.  Thus,  it  was  an  unexpected  finding  that  counselees  (partic- 
ularly the  youngest  ones)  evaluated  the  32-34  age  counselor  group  higher  than  the 
21-31  age  group.  Whatever  reason  might  account  for  this  result,  the  data  do 
Identify  the  counselor  age  group  which  the  youngest  counselees  (i.e.,  those 
probably  most  In  need  of  counseling  and  most  responsive  to  good  counseling) 
evaluated  as  most  effective  (on  criterion  Item  8 — see  Tables  7 and  8,  and 
Figure  5). 

The  finding  that  the  low  BTB  counselees  evaluated  counselors  as  more 
helpful  than  did  the  high  BTB  counselees  (see  Figure  4)  could  be  considered  an 
expected  and  desirable  result.  If  It  Is  assumed  that  the  low  BTB  counselees 
are  more  in  need  of  (or  would  benefit  more  from)  the  help  than  would  the  high 
BTB  counselees. 

Usefulness  of  Obtained  Validities 


The  usefulness  of  the  cross-validities  that  held  up  at  an  acceptable  level 
of  significance  (e.g.,  £<  .05)  may  be  demonstrated  by  application  of  the 
Taylor-Russell  (1939)  tables  for  various  selection  ratios  (see  Table  12).  For 
example,  if  the  applicant  pool  for  the  Navy  counselor  rating  contains  enough 
petty  officers  to  apply  a 50  percent  selection  ratio,  use  of  the  BAI  and  ICAC 
keys  as  a two-factor  selection  measure  would  increase  the  percentage  of  supe- 
rior counselors  selected  from  50  to  63  percent — a proportionate  Improvement  of 
26  percent.  Even  with  the  use  of  a standard  scale  with  a relatively  low  valid- 
ity, .22  for  the  CPS  Activity  scale,  and  a requirement  to  accept  most  applicants, 
70  percent,  some  Improvement  can  be  achieved — from  50  to  54  percent,  for  an  8 
percent  proportionate  Improvement. 


The  Individual  expectancy  chart  In  Figure  5 was  empirically  derived  from 
the  distribution  of  CPS  scores  of  the  cross-validation  sample.  The  base  rate 
of  58  percent  (see  Table  9 — CPS  high  criterion  group  N • 41)  represents  the  pro- 
portion of  Navy  counselors  found  presently  superior  (for  this  cross-validation 
sample)  without  the  use  of  the  CPS  as  a selection  device.  Specific  small 
samples  are,  of  course,  subject  to  sampling  errors.  Thus,  the  probabilities 
for  the  five-score  Intervals  In  Figure  6 do  not  present  a constant  progression 
of  probabilities  from  the  lowest  to  the  highest  score  Intervals.  To  reduce 
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the  effect  of  sampling  «rror,  Lawshe  et  al.  (1958)  developed  tables  to  con- 
struct theoretical  expectancy  charts.  Figure  7 was  constructed  by  Interpo- 
lating the  Lawshe  et  al.  tablular  data  for  a base  rate  of  58  and  r .34 — the 
parameters  for  the  results  of  the  cross-validation  of  the  CPS-constructed  key. 
With  the  use  of  this  chart,  an  applicant  who  scored  In  the  second  highest 
score  Interval  would  have  a probability  of  66  percent  of  performing  as  a 
superior  counselor — thus  eight  percentage  points  above  the  base  rate. 

It  Is  encouraging  that  some  of  the  cross-validities  did  hold  up  to  a 
level  that  permits  useful  applications,  since  the  distribution  of  counselee 
responses  on  the  criterion  Items  Indicated  that  most  counselors  were  doing 
a good  job  (see  Table  3).  Thus,  the  criterion  variance  Is  hardly  between 
good  and  poor  counselors,  but  rather,  between  superior  and  good  counselors. 

The  validities  obtained  may  justify  the  substantial,  additional  effort  that  was 
directed  towards  acquisition  of  criterion  data  directly  from  the  recipients  of 
the  services — the  counselees  (as  distinguished  from  the  usual  resort  to  acquisi- 
tion of  supervisory  evaluations) — and  using  multiple  raters  per  ratee. 

Although  the  keys  were  constructed  for  the  selection  of  career  counselors, 
they  may  be  useful  also  for  selection  to  other  occupations,  such  as  Recruiter, 
which  Involve  substantial  amounts  of  counseling  activities.  No  other  selection 
Instruments  known  to  the  authors  exist  that  were  empirically  validated  on  eval- 
uations acquired  directly  from  the  recipients  of  the  counseling  services — the 
counselee  or  the  recruit. 
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Table  12 

Improvement  in  Navy  Counselor  Performance  from 
^plication  of  Counselor  Selection  Instruments 


I 

i 


Instrument 
and  Scale 

Type 

Key^ 

Cross- 

Validity 

Selection 

Ratio 

Percentage  of  Identified  Superior 
Counselors  Improved^ 

From^ 

To  Absolute 

Increase 

Proportionate 

CPS-Activity 

Std 

.22 

30 

50 

59 

9 

18 

50 

50 

56 

6 

12 

70 

50 

54 

4 

8 

CPS-Navy 

Con 

.34 

30 

58 

73 

15 

26 

Counselor 

50 

58 

69 

11 

19 

70 

58 

65 

7 

12 

Multiple  BAI 

Con 

.41 

30 

50 

69 

19 

38 

and  ICAC-Navy 

50 

50 

63 

13 

26 

Counselor 

70 

50 

58 

8 

16 

Std- -Standard  Con--Constructed. 

'^Based  on  Taylor-Russell  (1939)  tables,  employing  the  tabled  r nearest  to 
the  obtained  cross-validation  r (e.g.,  for  CPS-Activity  r = .22,  tabled  £=  .20 
was  used) . 

Based  on  proportionate  size  of  the  high  criterion  subgroup  used  for  cross- 
validation  (see  Tables  9 and  10) . 
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CHANCES  IN  100  OF  BEING  A SUPERIOR  NAVY  CAREER  COUNSELOR 


Theoretical  individual  expectancy  chart  derived  from  the  constructed  COMREY  Personality 
Scale  Key  of  the  cross-validation  sample,  applying  the  Lawshe  et  al.  (1958)  tables. 


CONCLUSIONS 


1.  For  the  empirical  construction  of  keys  for  counselor  selection 
instruments,  it  is  feasible  and  most  appropriate  to  acquire  the  criterion 

data  directly  from  the  recipient  of  the  counseling  services — from  the  counselee. 

2.  Although  present,  nonquantitative  selection  procedures  produce  coun- 
selors who  provide  services  that  counselees  generally  evaluate  favorably,  the 
addition  of  noncognitive  instruments  to  the  selection  procedure  would  yield  an 
increased  percentage  of  superior  counselors  selected. 

3.  The  best  single  and  multiple  selectors  (and  their  cross-validities) 
were  the  constructed  keys  for  the  Comrey  Personality  Scales  (CPS)  (r_  * .34) 
and,  as  multiple  weighted  factors,  the  Biographical  Attltudinal  Inventory  (BAI) 
and  Ideal  Counselor  Adjective  Check  List  (ICAC)  (r^  = .41).  Although  either 

of  the  two  selectors  would  be  useful,  a choice  at  this  time  of  only  one  of 
them  would  be  premature,  since  the  size  of  the  cross-validation  sample  ’.jas 
small — for  the  CPS,  about  double  that  of  the  BAI-ICAC  composite.  Thus,  the 
obtained  significance  level  for  the  CPS,  £ < .006,  indicates  the  greater  con- 
fidence in  use  of  the  CPS,  even  though  the  cross-validity  of  BAI-ICAC  was  the 
greater. 


RECOMMENDATIONS 


It  is  recommended  that: 

1.  The  CPS  be  used  as  a selector  for  career  counselors. 

2.  The  BAI  and  ICAC  be  administered  but  not  used  as  multiple  selectors, 
and  that  all  three  instruments  be  revalidated  after  acquisition  of  additional 
data  to  provide  for  increased  sample  size. 

3.  The  usefulness  of  the  three  Instruments  for  other  jobs  involving 
counseling  activities  be  investigated. 
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